0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE
Computer Graphics and Applications
XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE 
Narrative Physicalization: Supporting Interactive 
Engagement with Personal Data  
 Maria Karyda  
Department of Design  
Aalto Universtiy  
Espoo, Finland 
maria.karyda@aalto.fi
Danielle Wilde  
University of Southern Denmark 
Kolding, Denmark 
d@daniellewilde.com 
         
Mette Gislev Kjærsgaard  
University of Southern Denmark 
Kolding, Denmark 
mgk@sdu.dk 
        
Physical  engagement  with  data  necessarily  influences  the 
reflective process. However, the role of interactivity and narration 
are often overlooked when designing and analyzing personal data 
physicalisations.  We  introduce  Narrative  Physicalisations, 
everyday  objects  modified  to  support  nuanced  self-reflection 
through  embodied  engagement  with  personal  data.  Narrative 
physicalisations borrow from narrative visualizations, storytelling 
with graphs, and engagement with mundane artefacts from data-
objects. Our research uses a participatory approach to research-
through-design and includes two interdependent, studies. In the 
first, personalized data physicalisations are developed for three 
individuals. In the second, we conduct a parallel autobiographical 
exploration of what constitutes personal data when using a Fitbit. 
Our  work  expands  the  landscape  of  data  physicalisation  by 
introducing narrative physicalisations.  It suggests an experience-
centric  view  on  data  physicalisation  where  people  engage 
physically with their data in playful ways, making their body an 
active agent during the reflective process. 
Keywords—data physicalisation, personal data, embodiment.  
I.  INTRODUCTION 
    Through  physicalisation,  digitalized  personal  data  can 
acquire physical properties [16], invite  people to “feel their 
data” [9], self-reflect and arrive at nuanced understandings 
about their data and themselves. To date, most personal data 
physicalisations  are  autobiographical  (e.g.  [1]).  In  such 
examples, a developer collects data, and through an iterative 
process,  synthesizes  and  constructs  their  personal 
physicalisation.  Baumer  et  al.  [2]  argue  that  to  achieve 
nuanced  reflection,  data  must  be  synthesized,  not  simply 
encountered.  The  aforementioned  process  affords  such 
synthesis  for  the  developer, but  not the  viewer. In  current 
examples of personal data physicalisation [1,16,19], data is 
encoded mainly in static representations. Being static limits 
the opportunity for self-reflection for the viewer, as it does not 
allow for ongoing synthesis of information.   
Our  research  examines  the  potential  of  full  body 
engagements, and narrative construction – storytelling – with 
personal  data  physicalisations.  We  hypothesis  these  two 
approaches may afford opportunities for people to synthesize 
their personal data and thus achieve nuanced self-reflection. 
To explore this idea, we developed Narrative Physicalisations, 
modified everyday objects that are interactive and designed to 
support self-reflection. Our concept builds upon the notion of 
narrative visualizations [14] and data-objects [19]. Narrative 
visualizations  are  traditionally  used  to  support  text  in 
journalism. They can facilitate narration of “data stories” and 
are  interactive.  In  complement,  data-objects  sit  at  the 
intersection of industrial design and data physicalisation. They 
resemble  familiar  artefacts,  and  can  thus  be  contextually 
situated. They benefit from the familiarity that people have 
with  everyday  objects,  yet  are  modified  to  incorporate 
personal data. 
In this article, we present two interdependent experimental 
studies.  The  first,  a  participatory  study,  unfolds  the 
development of three customized, narrative physicalisations 
of  participant data.  The second, an autobiographical study 
[10], was designed to challenge preconceptions around what 
constitutes personal data. Both studies use Fitbit devices to 
collect the data. Our studies extend the existing corpus of data 
physicalisation  research  by  introducing  the  concept  of 
narrative  physicalisations.  As  we  demonstrate,  through 
engagement  with  narrative  physicalisations  people’s 
reflective processes are enhanced.  This enhancement arises as 
the entire body is involved in deciding how personal data is 
synthesized. As a result, the data itself, depending on the way 
it is experienced, supports the narration of different stories.  
Our research examines: 
•  How  designers  and  developers  might  create 
physicalisations for dynamic bodily engagement with 
data? 
•  How  such  physicalisations  might  facilitate  self-
reflection,  and  what  the  role  of  narration  and 
storytelling therein might be?  
II.  RELATED WORK 
     Data physicalisation is experienced through the body and 
may become a ticket-to-talk in daily encounters facilitating 
reflection through bodily interaction and speech encounters.  
In the phenomenological view, as people become skilled 
with objects, the tools they use to enact each skill becomes an 
extension of their body. This phenomenon is what Clark and 
Chalmers call the extended mind [3]. The long cane used by 
a  blind  person  provides  a  much  discussed,  and  readily 
accessible  example [11]. Critically, the  feeling  that  a  tool 
becomes  an  extension  of  the  body  is  not  consistent. 
Heidegger [5], for example, argues that while nailing with a 
hammer,  the  tool  may  disappear  into  the  experience. 
However,  if  the  hammer  breaks,  it  suddenly  becomes  an 
object  in  the  world.  These  phenomena  demonstrate  the 
complexity of physical engagement with our surroundings. 
Indeed,  Sheets-Johnstone  argues  for  the  primacy  of 
movement in cognition [13] and Gallagher and Zahavi that 
the  body  literally  shapes  the  mind  [4].  These 
phenomenological  understandings  of  human  engagement 
with the material world are integral to our research. 
In data physicalisation, people experience data in different 
ways.  Barrass’s singing bowl [1] allows him to experience 
his blood pressure data through sound. This experience may 
Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore.  Restrictions apply. 

0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE

Computer Graphics and Applications

trigger unforeseen reflections. The most common senses

activated in data physicalisations are the “felt experience”

combined with sight [6]. The DNA ring [12], for example,

physically represents a person’s DNA. If the surface of the

ring is touched, the wearer can feel distinct data points. The

ring thus uses touch to engage with data. The ring is designed

to be worn on a daily basis, in real-world settings, where the

data may become a trigger for reflective discussion.

Similarly, Zhu et al.’s data-objects [19] explore the potential

of incorporating personal activity data into everyday objects

e.g. a mug with embedded data that can be experienced

through touch. In these examples, data physicalisations go

beyond their role as reflective object to incorporate dual

meaning. The objects suggest reflection through physical

properties, and may also be used to facilitate daily tasks. Such

objects become an “extension” of the user’s body through

familiar use. The data-objects become a ticket-to-talk and can

trigger a reflective process. However, the affordances of these

objects – that the data is fixed in time – does not allow for

deepened experimentation with the data.

In data-driven presentations when objects are taken out of

context, they can become facilitators to help people better

understand complex data. For instance, Hans Rosling

uses

everyday objects to explain complex data to various

audiences. He enriches this process by including aspects of

narration, achieving storytelling through the interaction with

the objects. In such data-driven presentations the nature of the

objects is not altered. Rather, the way the presenter interacts

with the objects alters their meaning. Through such data-

objects, self-reflection can become part of everyday life, and

interactivity and storytelling enhance people’s reflective

process. These elements remain underexplored in personal

data physicalisations.

III. STUDY DESIGN

Our research examines ways of combining activity data

and reminiscence to develop personalized data

physicalisations as prompts for self-reflection. We used a

participatory approach to research-through-design (pRtD)

[18] in two interdependent studies, over 9 weeks (Figure 1).

Study-1 began with semi-structured interviews of three

participants with differing interests. A Fitbit device was then

given to each of the participants to wear, for five weeks.

Hans Rosling. 2014. TED: Hans rosling’s ted talks.(2014)

Throughout the data-gathering, we followed participants’

data via the Fitbit software interface. We did not make

contact with them during this time. Our aim was to identify

whether participants’ physiological data would make evident

their unique lifestyles. After five weeks, a generative session

was held with each participant. These sessions used

physicalisations to make sense of participants’ personal data

and to speculate on “future data”.

Study-2, conducted in parallel (Week 1–5), was an

experimental autobiographical exploration in which the first

author (M) and a colleague (C) shared a Fitbit device,

exchanging it at weekly intervals. At the moment of exchange

M and C collaboratively reflected on the previous week’s

data. After three such discussions, they held a generative

session to co-construct physicalisations to reflect on their

data. This session concluded Study-2 (Timeline: Figure 1).

Drawing on both studies, we developed a single

physicalisation for each of the three participants in Study-1.

These were modified everyday objects with which the

participants were intimately familiar. We hypothesized that

the familiarity of the objects and mastery of use would render

the proposed interaction intuitive, and allow participants to

focus on their data.

Critically, the purpose of Study-2 was to gain first-hand

insights into the use of Fitbit, better understand and

empathize with participants, experience first-hand

physicalisation of personal data, question and challenge the

role of ‘the person’ in data generation and data

physicalisation, and thus enrich our engagement, as

researchers, with our Study-1 participants. As with any

autobiographical exploration, this process challenges

subjectivity and influence on a project [10]. However, as the

research was examining personalized interaction with

personal data, we determined the insights afforded would be

invaluable. Indeed, Study-2 enabled us to reflect and

iteratively test design ideas—responsive to, yet independent

of, Study-1—as the research evolved. This research design

aligns with the pRtD approach. It enabled us to deepen our

exploration, while gathering physiological data; during

generative sessions; and while prototyping physicalisations.

It allowed room for reflection during the design process, as

we hermeneutically cycled through the emerging findings—

Figure 1.The research timeline showing the overlap in the two studies.

Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.

Computer Graphics and Applications

interpreting and reinterpreting our understandings of what

personal data is, and how we might design for it.

To make the interdependencies between the two studies clear,

we present insights from each research activity at the end of

each section. We hope the documentation will allow other

researchers to reproduce our process. However, as with all

research through design, there is no expectation that others

following the same process would produce the same or even

a similar final artifact [20].

IV. RECRUITING PARTICIPANTS

Activity trackers serve a variety of industries from

commercial to pharmaceutical. That suggests a wide

spectrum of users, from patients to fitness enthusiasts and

others who are simply curious. We therefore sought

participants representing different industries and self-

tracking interests.

STUDY-1: We recruited three participants via personal and

professional contacts and social media, Ke, Ku and Mi. We

searched for people not actively self-tracking but with some

experience of wearables; and with different professional

backgrounds and interests.

STUDY-2: Involved the first author (M), who had no prior

experience with self-tracking; and C, a Fitbit user, recruited

through personal connections. C’s prior knowledge and

experience with wearables enabled us to track how the study

engaged previously held opinions about self-tracking.

We use pseudonyms for our participants to enhance the

personal tone in our narrative, as our work focuses on the

idiosyncratic nature of person-hood. Following, we

chronologically unfold our studies, alternating between them

where needed. This approach reflects the intertwined and

unfolding nature of the research activities.

V. STUDY-1: INTERVIEWS, PERSONAL DATA AND PRIVACY

Week 1, we conducted semi-structured interviews with Ke,

Ku and Mi. Our aim was to learn about their habits, interests,

important life experiences, motivation to participate in the

project, their future goals and views on what constitutes

personal data. We also aimed at establishing a context for

participant data collection which would then help us make-

sense of the gathered information during the project. The

three participants were in very different life moments. Ke, a

base player, had just graduated from a master’s degree and

was seeking employment; Ku, a computer programmer, was

struggling with demanding university courses; Mi, a retired

cancer caregiver, would be in India during the data gathering

period, receiving treatment to release tension.

During the interviews, we gained our participants’

informed consent for having their Fitbit data collected and

observed for a month. Then, each participant was given a

Fitbit and shown how it works. From that point, for five

weeks, we followed their data without establishing contact.

We observed their data graphs online and made our own

connections between the data and the information shared

during the interviews.

VI. STUDY-2: REFLECTIVE DISCUSSIONS

As noted, (above and in Figure 1), in weeks 1–4, M and C

shared a Fitbit. C had experience using the Fitbit as a personal

tracker and embraced its transformation into a collective

device.

At the beginnings of weeks 2–4 we held shared reflection

sessions. During these sessions, we used the online graphs

provided by FitBit as reminders of what had happened on any

day. The graphs helped M and C narrate their individual

experiences as if they were pictures of the place and the time

where data was generated. e.g. M (pointing to the graph):

“But do you know what happened [here]? I am going to show

you.” Notably, we found that the Fitbit data was sometimes

misleading. On a day where M achieved a large number of

steps, the device “assumed” based on her performance that

she was doing well. Whereas, M reflected that she was in an

emotionally difficult space on that day; her steps reflected

anxiety and frustration. This information nicely illustrates

how context was vital in making sense of the data.

Numerical data offers new perspectives, which can

motivate a person to change their behavior [9]. We found

that the additional factor of connecting another person with

the data helped motivate C to achieve or change their goals.

C: “When I had to do 600 steps I just started playing with my

cat and ran just to give [M] 10.000 steps. When [the device]

vibrated I went to bed”. This example demonstrates how

motivation can shift from achieving 10,000 steps for personal

reasons into achieving them for the other person. What

motivated C the most to do the remaining 600 steps was that

M would observe the graphs the following day. Having to

discuss whether or not she had achieved her personal goals

proved to be a more powerful motivator for C, than setting

her own goals and accounting only to herself. The human

factor thus shaped the device’s impact and on occasion

enabled C to strengthen their motivation through the device.

This is further evidenced in M and C’s online chat: C: “Since

you are tracking me, if I have [the device] only for one week

it is funny because it is both annoying and interesting. I have

been checking it more often because I want to know if I have

these 10,000 [steps] or not so you can see that I have made

it. ” And: “Sometimes I really considered to take a walk.” The

synergy between the human factors and the device helped C

reflect on her data, as well as her relationship with the device

itself. This reflection was afforded through her engagements

with another person (M). C’s behavior spoke to an idea of

personal data that goes ‘beyond the personal’.

Several times, C and M identified which graphs belonged

to each other without looking at the dates. C’s data graphs

had a similar visual flow every day and were identified with

ease. In contrast, M’s graphs had a variety of patterns that

made it difficult to identify what happened, and during which

day. As C explains to M: “[your graphs] look like fifteen

different people.” That indicated that there is a distinct

‘personality’ to the data, even if we need the context to make

sense of them and even if they are interpersonally generated.

A. Insights from the reflective discussions

The reflective discussions above, demonstrate that the

graph formats can constitute tickets-to-talk about the lived

experiences behind the data. Further, the graphs can be used

Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.

Computer Graphics and Applications

to support people’s memory. Through reflective discussions

we learn about C’s data, as well her personal life. These

findings indicate that graphs can support narration, essential

for understanding the data. We explore this possibility in

Study-1: Our Study-1 participants’ lifestyles were very

different from each other. These differences were evident in

their graphs. After two weeks of observations it was possible

to identify which graphs belonged to which participant by

sight alone. At times, we saw nuances in the patterns, but they

were still recognizable. We determined that these nuances, in

combination with the graphs, constituted tickets-to-talk, and

used them to design a customized generative session with

each participant. The aim was to learn more about their

activities during the data gathering period.

VII. GENERATIVE SESSIONS

In week 5, we conducted individual generative sessions

with our Study-1 participants to scaffold qualitative

information about their lived experiences during the data

gathering period. Following, we held a generative session in

Study-2, to understand how to leverage the insights from such

sessions when developing data physicalisations. We begin

with Study-1.

A. STUDY-1: Generative Sessions with Ke, Ku and Mi

Each generative session was divided in three parts: i)

we conducted interviews seeking to understand the

participant’s personal experience with the Fitbit, and their

opinions about being part of the project; ii) their graphs were

collated and used to support the participants to recall

experiences, which were discussed in detail; iii) participants

were introduced to two interactive props for physicalizing

data (Figure 2a). They were invited to choose one and, based

on memory, physicalize an average daily graph of the past

month, then speculate on a personal future graph and try to

physicalize that. We present our findings:

In the first task, we asked participants about their

experience of using the Fitbit device. Ke and Mi explained

that their behavior was influenced by the device. In contrast,

Ku reported that the device did not motivate him to be more

active. Ke referred to a moment where he had completed

“9000 something steps” and to achieve his daily target of

10000, started walking around his room. Ku said he only

looked at his daily data following a stressful experience, as at

these moments he was curious about his performance. Ku:

“It was after the fact! I was checking the result from the day

before” ... “For different instances I would check different

things. If it was a stressful day, I would check the pulse.”

Noticeably, while Ku said he was not influenced by the

device, he also said that his goal was to provide us with

“real” data. Ku: “I wanted you to get the actual data of how

I live but it was also a conscious decision not to change it.”

Notably, during the data gathering period Ku reduced his

alcohol consumption so that we would get ‘the actual data’.

Discussing a big night out, he explained: “I feel guilty

because of this all night out.” From his discussions, we could

see that he was influenced and motivated by the data in a

similar way to C. The need to share his lifestyle with us,

influenced his actions.

In terms of motivation, Mi explains: “It did make me walk

more but not as much as if I had been at home.” What was

obvious in our interview with Mi was that the data she

produced in India did not correspond to how her data would

have been if she was in Denmark. This topic was brought up

repeatedly throughout the generative session.

During the second task, our participants selected one or

two days and annotated their graphs with what they could

recall and discussed their lived experiences. In Mi’s case, the

main topic was the different lifestyle she was introduced to

while being in India compared to when she is in Denmark.

Mi: “What I found interesting was the heart rate going down

while I was in India.” Mi’s resting heartbeat graph illustrated

that her heart rate average was lower while abroad. This

outcome, justified to her the effectiveness of her treatment in

India. Mi: “This is not the real me” she said, suggesting that

the data we were following was not a realistic data picture of

herself, as it corresponded to her wellness retreat in India,

where she was advised to reduce her walking for her

treatment.

The most significant event for Ke during the data gathering

period was receiving two employment offers on one day. He

could recall and describe this day in great detail. His narration

included receiving the job offers, sharing this information

Figure 2. Generative session, task 3: (a) presenting the two

probes that assist people to physicalize their data; (b) Ku interacting

with one of the two probes.

Figure 3. Three participants during the generative sessions.

Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.

Computer Graphics and Applications

with his family and celebrating with friends. During the

celebration he consumed a lot of alcohol which resulted in a

high heart rate. Fitbit perceived this as exercise and

congratulated Ke with a virtual reward. Only when Ke

contextualized the data collection – alcohol consumption –

were we able to make-sense of his personal data.

The significant moment for Ku was a live performance

with his band. Before and after the concert his heart rate was

very high. He marked on the graphs when his heart rate was

high due to stress and when due to excitement. This again

surfaced a contradiction between the data graphs and the

lived experiences. Both heartbeat rates before and after the

concert were high. However, one peak corresponded to a

stressful moment, the other, with excitement or pleasure.

In the third part of the generative session (Figure 3), the

graphs that Ke and Mi physicalized were very similar to the

visualizations of Fitbit. In contrast, Ku (Figure 2b) focused

on narrating a story rather than creating a physicalisation that

would resonate with his graphs. The physicalisation he

developed helped him to share his story which again brings

the focus on aspects of narration and storytelling when

interacting with the graphs.

The most important finding of these generative sessions

was the ways that the experiences of participants linked to

their personal data. The interviews supported our premise that

data may partly reveal insights but is not able to demonstrate

the complexity and richness of an experience.

B. STUDY 2: Generative session M and C

The generative session with M and C served to explore

opportunities that data physicalisation may offer the

development of physicalisations for Study-1. The session was

divided into three parts. The first investigated the potential of

undated data cards (graphs) as reminders of specific events.

The second and third examined ways of co-creating

physicalisations based on those graphs.

In part 1, C was presented with three different graphs.

These graphs corresponded to the days of the month that the

shared Fitbit was handed over, and thus combined the data

from both C and M. The graphs were not dated; C was asked

to guess which days of the month were depicted on the cards.

We aimed to discover: i) if C was able to identify her personal

data; and ii) if she was able to recall a particular day simply

by looking at a graph. Notably, she characterized the data that

did not belong to her as faulty as she could not relate her

lifestyle to it.

Part 2 involved M and C working together to physicalize

the graphs discussed in part 1. The aim was to better

understand the role of materials when physicalizing a graph.

The first physicalisation was constructed from two pieces of

wood, each with 12 holes, and two roles of white and red

thread: one color for each person’s data-set (Figure 4b). The

artefact was painted white to represent a blank canvas. To

begin this exercise, C was asked to associate values to each

component of the physicalisation. She associated the top rod

to steps, explaining that the holes represented different

amounts of steps; and the bottom rod to time, and the shafts

to hours of the day. The white thread represented C’s data,

The red, M’s. The sole purpose of the different color threads

was to distinguish between the two datasets. The resulting

physicalisation shows one side in red and the other in white;

one side represents data from the morning, the other from the

evening (See Figure 5).

In part 3, C was asked to associate values to four differently

colored water glasses. Syringes were provided to use as a pen,

to ‘draw’ in color on a large paper sheet. The paper was

intentionally chosen such that the colored water would bleed,

and be somewhat uncontrollable. C associated blue with bad

feelings, red with good; black with time; and yellow with

activity (See Figure 4c).

At the end of the session, M and C analyzed the two

physicalisations. C proclaimed the first too abstract, as she

was not able to understand or read the graphs. To engage

reflectively with such an abstract physicalisation, a person

must learn to read it. The lack of specificity in the data and

the absence of what C felt were relatable material choices

seemed to eliminate any sense of the personal. C described

the resulting physicalisation as ’distant’. For the second

physicalisation, C explained that the way the color spread

provoked reflections about the nature of emotions and their

indeterminate form (Figure 4c). C: “The fact that it flows on

its own direction gives you the feeling that ok I can drop some

in many places so it makes it more real.” In this case, that

the physicalisation was abstract, less ‘precise,’ open for

interpretation, is what made it more ‘real’ for C. It allowed

for complexity and multiple interpretations of data. It was

Figure 4. l-r: a) laser cut activity graphs b) two-rod physicalisation c) physicalisation co-constructed by M and C using liquid colors.

Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.

Computer Graphics and Applications

deemed to come closer to ‘reality’ than a ‘fixed’, precise and

final-seeming form.

Because of these attributes, the second physicalisation

became a tool for reflection. It presented alternative image of

the participant’s self and prompted her to explore. New

relations between emotions and data, reality and extraction of

reality, were demonstrated here. This activity showed us that

it was possible to physicalize personal data in ways that

would add some of the richness of lived experiences.

Critically, the level of abstraction in the first physicalisation

seemed to depart too far from C’s experience. Whereas the

abstraction in the second—created by C, herself—supported

the reflective process.

Overall, the autobiographical exploration showed how a)

visualizations of personal data can support remembering, and

b) for reflective physicalisations the format should support

subjective expression.

The choice of the forms and materials used in the

generative session were not as rich as previous studies where

a variety of materials are provided for people to engage in

open ended ways in prototyping 7,16]. However, similar to

[7], our design constraints afforded more “focused

engagement with physicalization construction.” We

anticipated that a wider range of materials would make the

construction process more laborious. The two

physicalisations included abstract materials with diverse

properties. The aim was to cover different form and material

behaviors, and afford nuanced ideas and reflections. For

example, the physicalisation with two rods is a closed shape

which allows for limited extrapolation, while in the second

physicalisation, the colored water bleeds in ways that are

unpredictable affording nuanced reflections and insights.

VIII. STUDY 2: NARRATIVE PHYSICALISATIONS

To develop personalized physicalisations for Study-1, we

scheduled a two-week ideation phase. We reflected on the

themes that arose in the two studies and considered how to

highlight those themes. The strongest theme was that data is

always situated within a particular type of experience.

Two other criteria came out of the ideation phase: the

physicalisations i) should support data expression

through ambiguous formats, ii) should lay the ground

from which to recall lived experiences.

Based on these criteria, a personalized physicalisation was

developed for each participant. The participants’ character

traits also served as inspiration. We developed a foosball

table for Ke, a one string instrument for Ku and a treadmill

for Mi. Each of these included the data graphs in a unique

way (Figure 5). We invited our participants to interact with

these modified, interactive data objects. During the

interaction we asked open questions such as: “How did you

feel interacting with the object? Can you demonstrate how

did you feel that day?” (the day that the data was captured).

This approach aligns with Sanders and Stappers [15] work on

generative sessions. Following present the physicalisation

and testing.

A. The Foosball Table

Ke is a problem solver and a competitive person, who loves

foosball. We converged these traits to make him a personal

data Fusball Table (Figure 6). Our aim was to provoke him to

‘compete’ against his own data. The physicalisation had the

format of an ordinary foosball table with small modifications.

The handles of one opposite side were removed, so it could

only be played by one player. The remaining four handles

were made of wood and had a different number of foosmen

on each, so their weight varied. Handling the rods thus

required a specific learning of force and speed. A ping pong

ball was used to increase the level of difficulty and two fans

were placed closed to the net with the aim of blocking the

participant from scoring a goal – the fans blow the ball away

from the target and so make scoring more challenging.

The end of the table was left open to allow a data-sheet to

rotate freely on the table’s surface, like a conveyor belt

(Figure 6). The data-sheet showed the heartbeat of three

graphs: one from the day that Ke received two employment

offers, one from the day before, and one from the day after.

These graphs were chosen to evidence his behavior change

on the day he received the employment offers. The rules of

the game were: a) keep the ball within the borders of the

paper, b) do not let the ball fall out of the foosball table, c)

Figure 5. The completed two-rod probe (study 2).

Figure 6. The Foosball table.

Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.

Computer Graphics and Applications

score by the end of each day (meaning the period of time that

a graph would need to start appearing from the net side until

the moment that it would disappear completely from the

surface of the table).

At first, Ke identified the differences and similarities of the

physicalisation with a traditional foosball table, saying that

the game was different but the elements were the same. The

process of learning how to use it and then score took a while,

as the physicalisation had different affordances than a regular

foosball table. Ke first interacted with the artefact without

activating the rotation of the paper. Once the rotation was

activated, Ke was asked to follow the rules.

We observed that the graph did not seem to interfere with

the experience. Ke later demonstrated how he would play one

of the visualized days. The graphs did not assist him to

remember or tell his story; rather, his narration depended

completely on his memories. Though Ke did not use the

graphs in the physicalisation, the rotating paper added a level

of difficulty to the experience. “It is annoying that the paper

is moving because I can’t rest.” His experience demonstrates

the asymmetrical relationship between the embodied

familiarity Ke has with a regular foosball table compared to

his interaction with the physicalisation. The only moment the

graph played a prominent role was when Ke attempted to

follow the line of the visualization. This action took the

experience far from the initial concept. At this moment, Ke’s

vocabulary was influenced by his actions. Even his speech

was shaped by and adjusted to what he was doing. “It was

kind of all over the table” he said, referring to his mood,

instead of saying all over the place.

While interacting with the foosball, Ke’s sense of the data

passed into the background. The physicalisation failed to

trigger competition. Nonetheless, it led to a rich discussion

with Ke. We suggested making the visualizations more

prominent in the interaction and asked him to speculate how

he might interact with that physicalisation. Ke responded:

“The same agent is doing all these but has different wishes”

meaning that the person who generates the data also wants

the data to be easy to play to then win the game. The same

person could also change his lifestyle to win but with the cost

of making life compromises with a risk of a positive or

negative outcome. Reflecting on the use of the

physicalisation Ke said: “I am creating the difficulty while

living my life”.

The physicalisation is not trying to digest the ‘data’ for Ke.

Rather it seeks to provoke reflection and critique by showing

how our engagement with data is not ‘neutral’. Data may

reflect an experience (or aspects of it) and may spark stories

and reflections upon that experience. Ke’s last remark

indicates that physical interaction with data has the potential

to change not only the experience of the data, but the person

engaging with it.

B. The One String Instrument

Our second participant, Ku, has contradictory traits in his

personality – he is at once an introvert and a very social

individual. Ku was initially against the use of the tracking

device. We therefore aimed to develop a physicalisation that

would be pleasing for him to interact with. Fortunately,

finding what he was passionate about was easy as he is a

musician, a base player.

After working through several design ideas, we created a

one-string instrument (Figure 7). This design decision made

our design less complex than a multi-stringed instrument, yet

still engaging. The physicalisation consisted of one guitar

string, slightly elevated from the rest of its components. A

data-sheet of heartbeat visualizations slid under the string.

The heartbeat visualization was of four days during which Ku

was rehearsing and then performing in a concert. We

positioned the string against his heartbeat visualizations to

create a sonic asymmetry that made it difficult to generate a

pleasing sound. A plectrum and a glass container with a

cylindrical form were included to compliment the

physicalisation. Ku was asked to pluck the string with his

right hand and follow the line of the heartbeat data with the

glass container while the data-sheet slid under the string. He

thus could generate the soundtrack from each of his days.

Ke’s physicalisation is not dissimilar to a slide guitar and

was perceived as such. Ku has experience playing slide

guitar. This experience reinforced his excitement about

interacting with the physicalisation. His previous (embodied)

knowledge of playing string instruments helped him interpret

the graphs quickly. The way that the string was plucked in

parallel to the narration of his experiences seemed effortless.

Without being asked, Ku started following the line of the

graphs with the glass container, while plucking the string.

Unlike the foosball physicalisation, the data-sheet in this case

rotated from the beginning of the experiment. Ku: “The first

thing I want to do is to follow the line.” He described the

sound experience as almost disturbing or “messy.” After a

while, he developed an understanding of the sounds related

to high or low peaks. He explained, because he is a bass

player, he enjoys lower sounds. He was enthusiastic about the

lower peeks relating to him being calm.

Figure 7. The one-string instrument.

Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.

Computer Graphics and Applications

As with Ke and the foosball physicalisation, Ku was asked

to imagine and freely play the sounds of the days that were

on the data-sheet, based on his lived experiences. In

particular, we asked: “Can you play how those days felt?”

Here, a shift occurred in his interaction with the artefact.

Plucking the string and giving a high or low tempo according

to each day became more important to him than controlling

the sounds with the glass container. At a certain point, he

perceived the four graphs as a singular sonic experience. He

connected the high tempo to the excitement or the stress of

the days in question.

Similar to Ke’s experience with the foosball table, Ku’s

vocabulary was influenced by his interaction with the

physicalisation. He used the high tempo to describe the

intense days or, as he described them: days with pleasant or

unpleasant stress. Similar to C, his interaction with his

personalized physicalisation prompted him to share more

about the days of the graphs. This contrasted with his

reluctance to share during the generative session.

Ke’s physicalisation was deemed to have a ‘personality’

that provided context for making sense of the personal data.

Similar to Ku’s experience with his personalized data

Foosball Table, Ku’s sense-making was described in relation

to the interaction with the data, rather than with the data itself.

C. The Treadmill

While in India on her stress-relief retreat, Mi, our third

participant was instructed to reduce walking—one of her

favorite activities. This compromise transformed her data

image. It presented a strong contradiction between her

impression of her data in Denmark versus in India, and

inspired her personalized physicalisation. During ideation,

we focused on Mi’s step visualizations. When walking one

proceeds by advancing the feet alternatively so there is

always one foot on the ground. Steps unfold in a corrugated

way. When juxtaposed to graphs of steps, the way the action

unfolds in the body does not seem to match the graphs. The

differences between the visual output of a graph and the way

the action of walking looks, were significant elements of our

ideation process. This difference led us to wonder how one

might walk if one needed to follow the image of a graph.

To make the physicalisation, we modified a treadmill

(Figure 8) to fit the design idea. The top part was transformed

into a speech roster, including a stand on which the graphs

could lie. Six non-consecutive graphs were chosen for this

activity. Two, from the days with the lowest and highest

values of steps. The others were discussed during the

generative session. All six were placed on the platform in a

specific order, building from the day that Mi walked the least

steps to the day with the most steps. A square acrylic sheet

with the same dimensions as the graphs was placed next to

them. The sheet showed a color scale of eight colors from

dark blue to light pink. It was to be placed on the graphs and

divide them into the eight differed colors. At the bottom of

the treadmill was a wooden square, hiding a projector. The

projector was connected to the treadmill. As Mi walks on the

treadmill she generates a projection of the color scale on a

large wooden canvas on the opposite was. The colors change

from dark blue to light pink, according to Mi’s walking speed.

As she walks, Mi recalls and narrates her experiences.

The physicalisation was controlled by Mi, without sup-port

of an external power supply. The treadmill thus required extra

effort. Similar, to the other physicalisations, to interact

required learning how to use the device. While walking on

the treadmill, Mi said: “I had to use more force than what I

was expecting.” Her speed correlated with her ability to

identify the colors. On her second attempt, Mi tried to go

through the entire color scale; every time a color appeared,

she walked more slowly. Ultimately, she was able to identify

the changes between the colors, leading her to reflect: “I was

thinking that I often walk quite fast, and I like it, but I also

really like the slow. It is like you see more the changes.” This

reflection was facilitated by the affordances of the prototype.

After Mi learned how to use the treadmill, she focused on

looking at the graphs. She interpreted a large number of steps

as high speed and a small number of steps as low speed. She

explained: “When I look at this, is life in slow” meaning that

her experience in India shifted her actions from very active to

a slower pace. From then on, when describing experiences

during the testing she refers to them as slow or fast,

influenced by the interaction with the physicalisation. Then

her discussion transitioned to other life experiences. Her

reflections triggered questions such as, “How would you walk

your first wedding” or “How would you walk the birth of your

first child.” Walking was related to feelings. Her speed

reflected the overall feeling she had of an experience.

Occasionally when the feeling of the experience in the past

did not match the feeling of the experience in the present, it

was difficult for her to adjust her speed to the experience. The

conflict in her mind was evident in her bodily movements.

IX. DISCUSSION

All three physicalisations in Study-1 provided the

participants with a double opportunity. First to interact with

their personal data and relate to it through embodied learning

Figure 8. The treadmill.

Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.

Computer Graphics and Applications

(Figure 9). Second, through this learning, and by associating

new identities to their data, they were able to explore,

understand, share and reflect on their experiences, and

personal data more broadly. The overall interaction may be

divided in two parts:

A. Read and Act

The first type of interaction we observed was being guided

by the graphs but ignoring which day they represented. This

interaction positions the visualizations as the main actors of

the activity. The person interacts with their data yet does not

connect the data to an experience. Despite the gap between

data and experience, the interactions trigger reflections about

personal data on a different level. For instance, for Ku, the

lower sounds resonated with him being calm, this evoked

good feelings about the data. This experience was not

connected to any real-life experience captured by the data.

The associated interaction triggered a kind of private

knowledge. For both Ku and Mi, we observed the data

determine and lead the interaction as it became the element

of attention.

B. Interpret and Act

In this second phase of the interaction, each person had

completed reading their graphs and was trying to recall their

experiences based on the days depicted. Critically, the

physicalisations are customized artefacts that our participants

were intimately familiar with. The interactions, thus, benefit

from participants’ embodied knowledge. While interacting

with their physicalisations, the participants associate feelings

with their personal data. The graph seemed to provide an

“impression of data,” to play the role of a reminder. At times,

the personal data became transparent. For example, when the

participant focused on the interpretation of the experience

rather than on the data itself. The physicalisation thus became

a mediator through which the participants expressed their

feelings both bodily and through speech.

When the interpret and act interaction takes place, the

physicalisations become artefacts again in the world. They

become ready-to-hand tools that the participants know how

to use. The use of physicalisations in this way, leads to

reflections about people’s lived experiences. If we consider

that the participant’s body at once knows how to use an

artefact and is learning its use. The physicalisations thus have

a dual nature. They become the medium for interaction as

both master and student. The first nature is associated with

kinesthetic learning, the second focuses on emotions related

to lived experiences. Verbeek [17] argues that the mediating

role of technologies suggests two ways that artefacts can

influence humans being: as mediators of perception or

mediators of action. Artefacts become mediators of

perception when they mediate human experience and

interpretation of reality. e.g. technologies such as the

ultrasound shape the way we perceive and interpret a fetus

before it comes to the world. Mediation of action is related to

how artefacts incorporate meanings that provoke human

actions, e.g. a traffic sign signifies that drivers should slow

down, mediating this action through its material existence.

Similarly, data visualizations are mediators that do not

require an embodied relationship to provoke action. Instead,

they suggest action. Whereas, in personalized data

physicalisations, the graphs become ready-to-hand. They

transform into mediators of perception as participants, while

interacting with them, mentally build an interpretation of an

experience.

X. ON THE DEFINITION OF DATA PHYSICALISATION

Jansen et. al. propose that, “A data physicalization … is a

physical artifact whose geometry or material properties

encode data.”

Our physicalisations do not explicitly follow this definition,

as they incorporate people’s visualizations. The graphs were

included as necessary elements for people to recall the

experiences and reflect, highlighting the important of context

and experience to spark reflection. Our work extends

understandings of personal data physicalisation by

introducing interactivity; by illustrating the person-data

relation as dynamic, complex, ambiguous and unsettled -

reflected even when data remains ‘graphical’; and by

affording interaction with data, rather than simply

physicalisation of data.

Critically, in our personalized data physicalisations, the

graphs acted as triggers for interaction with the artefacts; by

extension with the data; and thereby narrative offerings from

the participants. Hogan et al. [6] argue that in dynamic

physicalisations the data is “encoded in the shape as well as

the behavior of the representation.” We extend that by

arguing that through bodily interactions with narrative

(interactive) physicalisations, the understanding of personal

data is extended. This extension allows the person to view

their personal human-data relationship as a whole rather than

as two separate elements. The data is encoded in the

interaction with the body and is extended by it.

XI. CONCLUSION

We presented a two-part study that resulted in three

personalized data physicalisations, what we call Narrative

Physicalisations. Our research demonstrates the potential in

narrative physicalisations to enable people to develop

meaningful narratives around data and personal experience.

Building on research into data objects, our findings

demonstrate that narrative physicalisations effectively

leverage the affordances of objects and the fact that people

know how to use objects. Following this work, designers and

developers of data physicalisations can use familiar objects

Figure 9. Ke and Ku interacting with the physicalisations.

Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.

Computer Graphics and Applications

to leverage participants’ embodied knowledge, and prompt

them to reveal insights that may not otherwise come to mind.

Our focus has been on people’s ability to engage with their

personalized objects and, through that engagement, recall

personal stories and reflect upon person-data relationships.

For future research, we anticipate exploring other sense

modalities for personalized, kinesthetic, tasty, smelly,

stimulating narrative physicalisations based on personally

meaningful data.

REFERENCES

[1] S. Barrass. Acoustic sonification of bloodpressure in the form of a

singing bowl. In Conference on Sonification of Health and

Environmental Data (SoniHED), 2014.

[2] E. P.S. Baumer, V. Khovanskaya, M. Matthews, L. Reynolds, V. S.

Sosik, G. Gay. Reviewing Reflection: On the Use of Reflection in

Interactive System Design. In Proceedings of the 2014 Conference on

Designing Interactive Systems, 2014, pp.93–102.

[3] A. Clark and D. Chalmers. 1998. The extended mind. analysis58, 1

(1998), pp. 7–19.

[4] Shaun Gallagher and Dan Zahavi. The phenomenological mind.

Routledge. 2012.

[5] M. Heidegger, J. Macquarrie, E. Robinson. Being and time, 1962.

[6] T. Hogan and E. Hornecker. Towards a design space for multisensory

data representation, Interacting with Computers, 2017, pp. 147-167.

[7] S. Huron, P. Gourlet, U. Hinrichs, T. Hogan, Y. Jansen. Let’s Get

Physical: Promoting Data Physicalization in Workshop Formats, In

Proceedings of the 2017 Conference on Designing Interactive Systems

(DIS ’17), New York, 2017, pp. 1409–1422.

[8] Y. Jansen, P. Dragicevic, P. Isenberg, J. Alexander, A. Karnik, J.

Kildal, S. Subramanian, K.Hornbæk. Opportunities and challenges for

data physicalization, In Proceedings of the 33rd Annual ACM

Conference on Human Factors in Computing Systems, 2015, pp. 3227–

3236.

[9] D. Lupton. Feeling your data: Touch and making sense of personal

digital data, New Media &Society19, 2017, pp. 1599–1614.

[10] C. Neustaedter and P. Sengers. Autobiographical design in HCI

research: designing and learning through use-it-yourself, 2012, pp.

514–523.

[11] M. Merleau-Ponty, Phenomenology of perception. London: Routledge,

(1945–2012).

[12] A. Rezaeian and J. Donovan. Design of a tangible data visualization.

In Proceedings of the 7

International Symposium on Visual

InformationCommunication and Interaction, 2014, pp.232–235.

[13] M. Sheets-Johnstone, The Primacy of Movement, John Benjamins,

1999.

[14] E. Segel and J. Heer. Narrative visualization: Telling stories with

data.IEEE transactions on visualization and computer graphics16,

2010, pp. 1139–1148.

[15] P.J. Stappers and E.BN Sanders.Generative tools for context mapping:

tuning the tools.InDesign and Emotion. Taylor & Francis London,

2003, pp.77–81.

[16] A. Thudt, U. Hinrichs, S. Huron, S. Carpendale. Self-Reflection and

Personal Physicalization Construction. In Proceedings of the 2018

CHI Conference on Human Factors in Computing Systems, 2018, pp.

1–13.

[17] P. Verbeek. Materializing morality:Design ethics and technological

mediation, Science,Technology, & Human Values31, 2006, pp. 361–

380.

[18] Wilde, D. Designing Research Education for Global Challenges. She

Ji Journal of Design, Economics and Innovation. Vol. 6, Issue 2

Summer 2020, pp. 170-212

[19] C. L. Zhu, H. Agrawal, P. Maes. Data-objects: Re-designing everyday

objects astactile affective interfaces, InternationalConference on

Affective Computing and IntelligentInteraction, 2015, pp. 322–326.

[20] J. Zimmerman, J. Forlizzi.Research Through Design in HCI. In: Olson

J., Kellogg W. (eds) Ways of Knowing in HCI. Springer, 2014,

Mary Karyda is a Doctoral Candidate in the Department

of Design at Aalto University, Helsinki. In her doctoral

research, Mary, explores meaningfulness - a part of

meaning making that relates to feelings - in relation to

physical representations of personal data in everyday life.

Her former training as an artist, in combination with her

background in design research inspires her to follow a

research through design approach. Her main research

interests revolve around data physicalization, personal

informatics, everyday objects and meaningfulness.

Danielle Wilde is Associate Professor of Embodied

Design at the University of Southern Denmark (SDU),

Kolding, where she leads SDU’s [body|bio] Soft Lab. She

specialises in participatory, speculative and critical research-

through-design, bringing focus to the social and ecological

impact of body-technology pairings and human-food

interactions. Core to her research, Wilde develops new

methods for thinking through moving, making and doing.

These methods enable diverse stakeholders to engage with

problems that cut cross disciplines and cultures, and develop

new practices, policies and technologies through a bottom-up

approach. Wilde publishes and exhibits widely across HCI,

Design Research, Citizen Science and Food Studies. 2020-

2021, she is Visiting Professor at Estonia Academy of Art

Doctoral School, helping them to build their research

capacity. For more, see: www.daniellewilde.com

Mette Gislev Kjærsgaard has worked with combinations

of design and anthropology in organizational as well as

academic contexts for the past twenty years. In 2011 she

received her Phd in Anthropology from the Department of

Culture and Society, University of Aarhus. She currently

holds a position as associate professor of design anthropology

at SDU design, University of Southern Denmark. Mette has

conducted research on i.a. community-based innovation,

digital play and design processes with a particular focus on

the applied and academic potentials of a design

anthropological approach. She is co-organizer of Research

Network for Design Anthropology and co-editor of Design

Anthropological Futures (Bloomsbury 2016).

Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.